Repository navigation
Conversation
verl-project/verl#7461 makes FusedLinearForPPO route the fused output head through Liger's LigerFusedLinearScaledCrossEntropyFunction whenever liger_kernel is importable. That kernel bypasses vexact's aten::mm override, so training-side log-probs stop matching inference log-probs bit-for-bit: test_fused_linear_for_ppo_log_probs_float32_in_bfloat16_model fails with 98.9% mismatched elements (max abs diff 0.6875) on verl 8e4a572 + liger 0.8.2. While batch-invariant mode is on, null verl's module-level dispatch handle so FusedLinearForPPO falls back to its chunked torch.matmul + cross-entropy path (which the overrides cover), and restore it when the mode is turned off. The verl integration imports verl's module before enabling the mode so the hook always lands, and enable_batch_invariant_mode() re-applies it when the mode was already on. Add a test for the toggle. Verified on 1xH100 with verl 8e4a572 + liger-kernel 0.8.2: tests/batch_invariant_ops/ 68 passed (was 2 failed).
Collaborator
|
please make PR to verl to avoid monkeypatch |
5 tasks done
wuxibin89
pushed a commit
to verl-project/verl
that referenced
this pull request
Sep 3, 2026
### What does this PR do? Makes `fused_kernel_options.impl_backend` authoritative for the PPO output head: - `torch` always uses verl's native chunked `FusedLinearForPPOFunction`. - `liger` explicitly opts into `LigerFusedLinearScaledCrossEntropyFunction` and retains the native fallback when Liger is unavailable. - The selected backend is propagated through all five Hugging Face fused-forward adapters. This fixes a regression introduced by #7461. That change selected Liger whenever it was importable, even when the configured backend was `torch`. Besides violating the backend contract, the implicit Liger dispatch bypasses PyTorch operator overrides used by exactness integrations such as vexact, forcing vexact to modify verl's private module state as proposed in verl-project/vexact#48. This does not duplicate an existing PR. Before implementation, open PRs and issues were searched for `FusedLinearForPPO Liger batch invariant`, `LigerFusedLinearScaledCrossEntropyFunction`, `fused linear PPO exactness`, and `disable Liger fused linear`; no matching open work was found. AI assistance: OpenAI Codex assisted with implementation and test execution. The human submitter reviewed every changed line and understands the change end-to-end. The relevant tests listed below were run in the shared development workspace before submission. ### Test - `python -m pytest -q tests/utils/test_experimental_torch_functional_on_cpu.py` — 6 passed. - `python -m pytest -q tests/models/test_fused_kernels_ulysses_sp_on_cpu.py` — 26 passed. - `pre-commit run --files <all changed files>` — all hooks passed. - vexact `test_fused_linear_for_ppo_batch_invariance_forward` on 1x H20, without a vexact monkey patch or environment override — 2 passed. Additional H20 diagnostic: the full vexact file has two existing strict bitwise comparison failures between the native chunked 2D matmul path and the 3D reference matmul, with maximum absolute differences of `1.52587890625e-05` and `1.9073486328125e-06`. This is separate from the implicit Liger dispatch fixed here; verl-project/vexact#48 reports 68 passing tests on H100 when using the same native fallback path. ### API and Usage Example The default remains the native torch backend: ```yaml actor_rollout_ref: model: use_fused_kernels: true fused_kernel_options: impl_backend: torch ``` Select Liger explicitly when desired: ```yaml actor_rollout_ref: model: use_fused_kernels: true fused_kernel_options: impl_backend: liger ``` ### Design & Code Changes - Add an explicit `impl_backend` argument to `FusedLinearForPPO`, defaulting to `torch`. - Route to Liger only when `impl_backend="liger"`. - Store the selected fused backend on each model instance and pass it through dense, Qwen2-VL, Qwen3-VL, Qwen3.5, and GLM4V torch forward adapters. - Document `liger` as an explicit backend and clarify the `torch` backend behavior. - Add unit coverage for backend dispatch, validation, and adapter propagation. ### Checklist Before Submitting - [x] Read the contribution and agent instructions. - [x] Search for duplicate issues and PRs. - [x] Add/update tests and documentation. - [x] Run pre-commit checks. - [x] Disclose AI assistance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
verl-project/verl#7461 makes FusedLinearForPPO route the fused output head through Liger's LigerFusedLinearScaledCrossEntropyFunction whenever liger_kernel is importable. That kernel bypasses vexact's aten::mm override, so training-side log-probs stop matching inference log-probs bit-for-bit: test_fused_linear_for_ppo_log_probs_float32_in_bfloat16_model fails with 98.9% mismatched elements (max abs diff 0.6875) on verl 8e4a572 + liger 0.8.2.
While batch-invariant mode is on, null verl's module-level dispatch handle so FusedLinearForPPO falls back to its chunked torch.matmul + cross-entropy path (which the overrides cover), and restore it when the mode is turned off. The verl integration imports verl's module before enabling the mode so the hook always lands, and enable_batch_invariant_mode() re-applies it when the mode was already on. Add a test for the toggle.
Verified on 1xH100 with verl 8e4a572 + liger-kernel 0.8.2: tests/batch_invariant_ops/ 68 passed (was 2 failed).
What does this PR do?
Checklist Before Starting
[{modules}] {type}: {description}format (see check_pr_title.py for the full list of allowed modules and types)[BREAKING]— e.g.[BREAKING][ops] feat: new batch-invariant matmul APITest
API and Usage Example
Design & Code Changes
Checklist Before Submitting
pre-commit run --all-files)